System, method and article for normalization and enhancement of tissue images

ABSTRACT

In medical imaging, a fiducial marker facilitates tissue image correlation that allows for image analysis, normalization and correction of the optical exposure and spectral and spatial distribution in order to compensate for the surface reflections, sub surface tissue interactions and spatial orientation of the excitation and imaging axes to the subject tissue. Using a cross comparison, clinicians can model tissue image data in different forms in order to reference and compare data from various spectral components and or from different images. This may enhance human interpretation between images including the variations between images even when the spectral, spatial and optical conditions or the image resolution or sensitivity are compromised. Such may be used to assess cosmetic, moisturizing, therapeutic materials and treatments.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. national phase entry of InternationalApplication No. PCT/US2011/027512, filed Mar. 8, 2011, which applicationclaims the benefit under 35 U.S.C. 119(e) of U.S. provisional patentapplication Ser. No. 61/311,750, filed Mar. 8, 2010, which areincorporated herein by reference in their entireties.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present disclosure relates to the field of medical imaging and inparticular to clinical imaging of tissue such as skin or other bodilytissue, with or without lesions, for reference and analysis.

2. Description of the Related Art

Conventional methods of clinical imaging employ methods where there islittle or no control over the light source, exposure, orientation to thesubject or the optical characteristics of the image. Images are used todocument the visible characteristics of a scene and are used in diversefields such as remote sensing, dermatology and forensics. In some cases,a measurement tool is introduced into the imaging field of view to allowfor approximate correlation of the captured images to a linear scale. Inphotogrammetry time scale images are compared and corrected for spectraland spatial frequency distributions. This is often a laborious process.Spectral artifacts are difficult to correlate in a time series ofdigital images due to variation in angles of the source light andvariation in optical axes and the impacts of ambient conditions.Photogrammetric observations use tracking of parameters such asposition, distance from the subject and time, to ensure the opticalangles of reference can be used in image correlation and rectification.Rectification is often complicated by the three dimensionalcharacteristics of the scene. In medical imaging, coordinate systems canbe used to spatially relate subject matter to a standard, such as theTalairach atlas.

Applicant is not aware of any standard by which photographic orspectroscopic images of human tissue can be used to repeatedly establishthe tissue specific molecular optical characteristics of all subjects atdifferent times, with different optical conditions. U.S. Pat. No.6,738,652 discusses the optical thickness of the skin as a ratio ofprotein to fat spectra. The correlation of an image time series usingconventional techniques is complicated by the inability to accuratelycompensate for exact changes due to variations in the ambientconditions.

Therefore there is a need for technical approaches in clinical imagingthat allow clinicians to quickly acquire skin images without having toconcern themselves with the complex optical considerations that surroundthe rectification and registration of images.

This background information is provided to reveal information believedby the applicant to be of possible relevance to the present application.No admission is necessarily intended, nor should be construed, that anyof the preceding information constitutes prior art against the presentapplication.

BRIEF SUMMARY

Systems, methods and articles allow correction of tissue imagescorresponding to spectral effects in a tissue sample due to the complexinteractions of light and where a computer model of tissue image datamay be used to cross reference and compare data from various spatial andspectral components and or from different images, to model lesion shape.

Systems, method and articles allow correction and analysis of a digitalimage in three dimensions, where physical and/or virtual fiducialmarkers are used in the imager field of view, and where the fiducialmarker is of a form that includes a well defined shape and colorvariations and where some of the color areas on the fiducial marker areused as optical phantoms to match the spectral character of livingtissue including a reflective layer to simulate the optical character ofthe skin.

A system and a method may be summarized as performing correction andanalysis of digital images in three dimensions in which a fiducialmarker appears, the fiducial marker having a well defined shape andcolor variations, where some of the color areas on the fiducial markerare optical phantoms to match the spectral character of living tissueand have a reflective layer to simulate an optical character of livingbodily tissue such as skin.

The correction may correspond to a set of spectral effects of the tissuesample, which arise due to complex interactions of light. A computer ordigital model of tissue image data may be used by the system or methodto cross reference and compare data from various spatial and spectralcomponents and/or from different images, to model lesion shape. Theanalysis comparison of layers may take the form of a histogram.

The system or method may use the optical spectral data from the digitalimage to create a three dimensional digital reconstruction of thelesion, including multispectral data and image timeline data.

The optical phantom can be related to the individual spectral componentsof skin with layers. The system or method may normalize a multiple imageseries. The optical phantom may include an Epidermal Layer Phantom inthe visible spectrum of 500 nm to 600 nm; a Melanin Layer Phantom in thespectrum of 300 nm to 500 nm; a Hemoglobin Layer Phantom whereoxyhemoglobin absorption peaks close to 415 nm, 515 nm, 590 nm or wherein contrast deoxyhemoglobin peaks at 430 nm, 575 nm, 610 nm; a CollagenLayer Phantom where absorption is between 340 nm to 400 nm with afluorescence peak between 450 nm to 550 nm (the fluorescent peak may becreated with fluorescein or 5-carboxyfluorescein), a water phantom whereabsorption peaks in the range of 450 nm; 575 nm; 630 nm; 730 nm; 820 nm,or in contrast reflection peaks at 514 nm; 606 nm; 660 nm; 739 nm.

A system and method for correction and analysis of digital images inthree dimensions, may employ one or more fiducial markers, which arephysically placed or optically projected into the image field of view,and where the fiducial marker is of a form that includes a well definedshape and color variations and where some of the color areas on thefiducial marker that are used as optical phantoms to match the spectralcharacter of living tissue also have a reflective layer to simulate theoptical character of the skin.

A digital image time series is normalized using numerical methods, forexample by measuring the difference of the spectral distribution betweenthe optical character of the tissue in combination with the fiducialmarker where the monotonicity of certain spectral relationships may beoutside or approaching the limit of the normal spectral distribution.Normal may be determined by the optical character of a subjects' healthytissue or in comparison to a population. A digital image may benormalized to the fiducial marker, yet require to be normalized based onthe spectral markers such those of hemoglobin and collagen. Anormalization of optical relationships may result in an analysisincluding generation of a probability distribution of a tissue beingabnormal. The abnormal relationship may be represented by an opticaldensity compared to a percentage of optical spectra that can beattributed to that of collagen. An abnormal relationship represented inthe digital images may be viewed within an assigned probability index,that allows certain digital images to be weighted by their diagnosticvalue or weighted by comparative changes between spectra, which may showor be indicative of a trend. The optical properties of hemoglobin,collagen, melanin and/or epidermis may be used as an optical signaturerepresentative of tissue or of a specific individual person.

Correction for correlation of digital images may also include colorcorrection information in order to establish lesion border parametersthat might include fluorescence and reflection changes, or by comparisonof the surface optical spectra to the subsurface spectra. Amultidimensional lesion map can be made to track the pixelcharacteristics in the digital image such as surface, sub-surface andother layers or depth characteristics as may be determined from thespectral analysis, such as areas of molecular activity, blood flow ortissue density. Variation between image layer coordinates in a timeseries of digital images may be used for registration and a standardizedmethod of optical correlation.

The optical or spectral data may be combined with spatial data to createtrue three dimensional digital models of the lesions including growthcomparisons, and where the shape of the tissue is rectified with a threedimensional topographic map of the body that combines three dimensionalmodel probabilities, with correlation of coordinate locations andspectral effects and complex interactions.

The normalized images enable clinicians through automated updates, to beable to make diagnostic decisions using a standard protocol. The systemand method may also be used to establish a baseline optical/molecularindex for an individual patient and this data used to contribute to thenormalization of images or be used in a timeline as a diagnostic test.In particular, the digital images may be analyzed and a probabilityindex created from the combination of distributed properties of thevariables including normalization, exposure correction, geometriccorrelation, optical spectroscopic correction, signal to noisecharacterization and diagnostic protocols.

A method of operating a system for use in tissue analysis may besummarized as including comparing by at least one processor anappearance of at least one shape of at least a first fiducial marker ina first digital image of a portion of a tissue to at least one definedactual shape of the fiducial marker; comparing by the at least oneprocessor an appearance of each of a plurality of sections of thefiducial marker in the first digital image to respective ones of definedsections of the fiducial marker including a number of tissue phantomseach having a respective spectral characteristic that matches arespective spectral characteristic of tissue of a type represented inthe first digital image; and at least one of correlating, normalizing,or correcting at least the first digital image, based at least in parton the comparisons. The fiducial marker may include a scatter layer thatoverlies at least some of the tissue phantoms and which simulates anoptical character of the type of tissue represented in the first digitalimage, and wherein comparing an appearance of each of a plurality ofsections of the fiducial marker in the first digital image to respectiveones of defined sections of the fiducial marker includes comparing theappearance of the sections which include the tissue phantoms which areoverlaid by the scatter layer with a number of defined sections whichinclude the tissue phantoms overlaid by the scatter layer. A number ofsections of the fiducial marker may include a respective color includingat least one of black, white, a plurality of different shades of grey,and a plurality of additional colors that are not black, white or grey,and wherein comparing an appearance of each of a plurality of sectionsof the fiducial marker in the first digital image to respective ones ofdefined sections of the fiducial marker includes comparing theappearance of the sections which include the respective colors withrespective ones of a defined set of respective colors.

The method may further include storing to at least one nontransitorystorage medium the digital image as a multi-layer image file, includinga first digital image layer that stores and at least a second digitalimage layer that stores image metadata.

The method may further include storing to a diagnostic layer of thedigital image on the nontransitory storage medium information indicativeof at least one of an NADH fluorescence, a collagen fluorescence, aphysical scattering of light from the tissue at a number of physicallayers of the tissue due to tissue density, a spectral distribution dueto a size of a cell nuclei, and a hemoglobin absorption due to increasedblood flow or oxygenation.

The method may further include registering a number of subsequentdigital images in spatial and optical relationship by the at least oneprocessor; and comparing the first and the subsequent digital images ona layer by layer basis by the at least one processor.

The system may further include referencing by the at least one processorat least one of spectral changes or optical density at specificcoordinates in the first digital image to allow later comparison tochanges in a number of subsequent digital images of the region ofinterest.

The method may further include comparing by the at least one processor anumber of ratios of respective radiant spectral intensity of a number ofwavelengths or wavebands in the first digital image.

The method may further include comparing by the at least one processor anumber of ratios of respective radiant spectral intensity of a number ofwavelengths or wavebands in at least one subsequent digital image.Normalizing may include normalizing a plurality of digital imagesincluding the first digital image by measuring a difference of aspectral distribution between an optical character of the tissue incombination with the fiducial marker, where a monotonicity of a numberof defined spectral relationships is proximate or exceeds a limit of anormal spectral distribution.

The method may further include establishing a subject specific baselineby the at least one processor which is specific to an individual; andwherein the normalizing is based at least in part on the subjectspecific baseline the first digital image and a plurality of sequentialdigital images, the sequential digital images sequentially captured atvarious times following a capture of the first digital image.

The method may further include determining a number of differences inthe region of interest as the region of interest appears between thenormalized digital images including the first digital image and theplurality of sequential digital images, by the at least one processor,as part of a tissue analysis. Determining a number of differences mayinclude determining any morphological changes of the region of interestas the region of interest appears between the digital images as part ofthe determination of the differences in the region of interest as theregion of interest appears between the normalized digital imagesincluding the first digital image and the plurality of sequentialdigital images. Determining a number of differences may includeassessing any change in at least one of a level of skin hydration, atotal number of wrinkles or a size of at least one wrinkle, or a totalnumber of blemishes or a size of at least one blemish. Determining anumber of differences may include assessing at least one of a level ofhydration or a level of blood flow between the first digital image andat least one subsequent digital image, where the first digital imagerepresents the region of interest prior to a first application of acosmetic, a moisturizer, a therapeutic or a therapeutic treatment andthe at least one subsequent digital mage represents the region ofinterest after the first application of the cosmetic, the moisturizer,the therapeutic or the therapeutic treatment. Normalizing may includenormalizing at least the first digital image based at least in part on aspectral marker of hemoglobin and a spectral marker of collagen.

The system may further include generating a probability index by the atleast one processor based on a combination of distributed properties ofa number of variables including a normalization, an exposure correction,a geometric correlation, an optical spectroscopic correction, a signalto noise characterization, or a defined diagnostic protocol. Theinstructions may further cause the at least one processor to generate adigital model that geometrically represents the region of interest inthree dimensions based on spatial and spectral data from the digitalimages.

The method may further include associating at least one of multispectraldata or image timeline data to the digital model that geometricallyrepresents the region of interest in three dimensions by the at leastone processor.

The method may further include rectifying the tissue by the at least oneprocessor with a three dimensional map of at least a portion of a bodywhich combines a set of three dimensional model probabilities with acorrelation of a set of coordinate locations, a set of spectral effectsand a set of complex interactions. Correcting may include correcting atleast the first digital image based at least in part on color correctioninformation.

The method may further include generating by the at least one processora digital multidimensional lesion map that tracks a set of pixelcharacteristics in at least the first digital image including at leastone of a surface, a sub-surface, other layers or a depth characteristicof the tissue as determined from a spectral analysis of the tissue asrepresented in at least the first digital image.

Correcting may further include correcting for spectral effects in thetissue represented in at least the first digital image which spectraleffects are due to interactions of light absorption, reflectance andfluorescence, and to cross reference and compare a number of spatial anda number of spectral components specified by at least one of a digitalmodel of tissue image data or another digital image to generate thedigital three dimensional model of the region of interest. Correctingmay include correcting for differences in spatial orientation of atleast one of an excitation axis or an imaging axis of a tissue imagingsystem in Cartesian space.

The method may further include registering each of a plurality ofdigital images of the tissue by the at least one processor, includingthe first digital image, based at least in part on a variation betweenimage layer coordinates in a temporal sequence of a plurality of digitalimages of the tissue.

The method may further include generating by the at least one processoran analysis comparison of layers in at least the first digital image asa histogram.

The method may further include generating by the at least one processora probability distribution of a tissue being abnormal. Generating aprobability distribution of a tissue being abnormal may includegenerating the probability distribution of the tissue being abnormalbased at least in part on a comparison of an optical density to apercentage of optical spectra that is attributable to collagen. Aprobability distribution of a tissue being abnormal may includegenerating the probability distribution with a probability index thatweights at least some digital images according to at least one of adiagnostic value or a comparative amount of change between spectra.

A system for use in tissue analysis may be summarized as including atleast one processor; and at least one nontransitory storage medium thatstores processor executable instructions which when executed cause theat least one processor to: compare an appearance of at least one shapeof at least a first fiducial marker in a first digital image of aportion of a tissue to at least one defined actual shape of the fiducialmarker; compare an appearance of each of a plurality of sections of thefiducial marker in the first digital image to respective ones of definedsections of the fiducial marker including a number of tissue phantomseach having a respective spectral characteristic that matches arespective spectral characteristic of tissue of a type represented inthe first digital image; and at least one of correlate, normalize, orcorrect at least the first digital image, based at least in part on thecomparisons. The fiducial marker may include a scatter layer thatoverlies at least some of the tissue phantoms and which simulates anoptical character of the type of tissue represented in the first digitalimage, and the instructions cause the at least one processor to comparethe appearance of the sections which include the tissue phantoms whichare overlaid by the scatter layer with a number of defined sectionswhich include the tissue phantoms overlaid by the scatter layer. Anumber of sections of the fiducial marker may include a respective colorincluding at least one of black, white, a plurality of different shadesof grey, and a plurality of additional colors that are not black, whiteor grey, and the instructions cause the at least one processor tocompare the appearance of the sections which include the respectivecolors with respective ones of a defined set of respective colors.

The instructions may further cause the at least one processor to storethe digital image as a multi-layer image file, including a first digitalimage layer that stores and at least a second digital image layer thatstores image metadata.

The instructions may further cause the at least one processor to storeto a diagnostic layer of the digital image information indicative of atleast one of an NADH fluorescence, a collagen fluorescence, a physicalscattering of light from the tissue at a number of physical layers ofthe tissue due to tissue density, a spectral distribution due to a sizeof a cell nuclei, and a hemoglobin absorption due to increased bloodflow or oxygenation.

The instructions may further cause the at least one processor toregister a number of subsequent digital images in spatial and opticalrelationship and to compare the first and the subsequent digital imageson a layer by layer basis.

The instructions may further cause the at least one processor toreference at least one of spectral changes or optical density atspecific coordinates in the first digital image to allow latercomparison to changes in a number of subsequent digital images of theregion of interest.

The instructions may further cause the at least one processor to comparea number of ratios of respective radiant spectral intensity of a numberof wavelengths or wavebands in the first digital image.

The instructions may further cause the at least one processor to comparethe number of ratios of respective radiant spectral intensity of thenumber of wavelengths or wavebands in the first digital image to anumber of ratios of a respective radiant spectral intensity of a numberof wavelengths or wavebands in at least one subsequent digital image.

The instructions may further cause the at least one processor tonormalize a plurality of digital images including the first digitalimage by measuring a difference of a spectral distribution between anoptical character of the tissue in combination with the fiducial marker,where a monotonicity of a number of defined spectral relationships isproximate or exceeds a limit of a normal spectral distribution.

The instructions may further cause the at least one processor toestablish a subject specific baseline which is specific to anindividual, and normalize based at least in part on the subject specificbaseline the first digital image and a plurality of sequential digitalimages, the sequential digital images sequentially captured at varioustimes following a capture of the first digital image.

The instructions may further cause the at least one processor todetermine differences in the region of interest as the region ofinterest appears between the normalized digital images including thefirst digital image and the plurality of sequential digital images aspart of a analysis.

The instructions may further cause the at least one processor todetermine morphological changes of the region of interest as the regionof interest appears between the digital images as part of thedetermination of the differences in the region of interest as the regionof interest appears between the normalized digital images including thefirst digital image and the plurality of sequential digital images. Theinstructions may cause the at least one processor to determine thenumber of differences by assessing any change in at least one of a levelof skin hydration, a total number of wrinkles or a size of at least onewrinkle, or a total number of blemishes or a size of at least oneblemish. The instructions may cause the at least one processor todetermine the number of differences by assessing at least one of a levelof hydration or a level of blood flow between the first digital imageand at least one subsequent digital image, where the first digital imagerepresents the region of interest prior to a first application of acosmetic, a moisturizer, a therapeutic or a therapeutic treatment andthe at least one subsequent digital mage represents the region ofinterest after the first application of the cosmetic, the moisturizer,the therapeutic or the therapeutic treatment.

The instructions may further cause the at least one processor tonormalize at least the first digital images] based at least in part on aspectral marker of hemoglobin and a spectral marker of collagen.

The instructions may further cause the at least one processor togenerate a probability index based on a combination of distributedproperties of a number of variables including a normalization, anexposure correction, a geometric correlation, an optical spectroscopiccorrection, a signal to noise characterization, or a defined diagnosticprotocol.

The instructions may further cause the at least one processor togenerate a digital model that geometrically represents the region ofinterest in three dimensions based on spatial and spectral data from thedigital images.

The instructions may further cause the at least one processor toassociate at least one of multispectral data or image timeline data tothe digital model that geometrically represents the region of interestin three dimension

The instructions may further cause the at least one processor to rectifythe tissue with a three dimensional map of at least a portion of a bodywhich combines a set of three dimensional model probabilities with acorrelation of a set of coordinate locations, a set of spectral effectsand a set of complex interactions.

The instructions may further cause the at least one processor to correctat least the first digital image based at least in part on colorcorrection information.

The instructions may further cause the at least one processor togenerate a digital multidimensional lesion map that tracks a set ofpixel characteristics in at least the first digital image including atleast one of a surface, a sub-surface, other layers or a depthcharacteristic of the tissue as determined from a spectral analysis ofthe tissue as represented in at least the first digital image.

The instructions may further cause the at least one processor to correctfor spectral effects in the tissue represented in at least the firstdigital image which spectral effects are due to interactions of lightabsorption, reflectance and fluorescence, and to cross reference andcompare a number of spatial and a number of spectral componentsspecified by at least one of a digital model of tissue image data oranother digital image to generate the digital three dimensional model ofthe region of interest.

The instructions may further cause the at least one processor to correctfor differences in spatial orientation of at least one of an excitationaxis or an imaging axis of a tissue imaging system in Cartesian space.

The instructions may further cause the at least one processor to performa registration on each of a plurality of digital images of the tissue,including the first digital image, based at least in part on a variationbetween image layer coordinates in a temporal sequence of a plurality ofdigital images of the tissue.

The instructions may further cause the at least one processor togenerate an analysis comparison of layers in at least the first digitalimage as a histogram.

The instructions may further cause the at least one processor togenerate a probability distribution of a tissue being abnormal.

The instructions may further cause the at least one processor togenerate the probability distribution of the tissue being abnormal basedat least in part on a comparison of an optical density to a percentageof optical spectra that is attributable to collagen.

The instructions may further cause the at least one processor togenerate the abnormal relationship of the images are viewed within aprobability index that weights at least some digital images according toat least one of a diagnostic value or a comparative amount of changebetween spectra.

A fiducial marker for use in tissue imaging may be summarized asincluding a substrate having a defined profile and bearing a pluralityof sections having respective wavelength selective absorption,reflectance or florescence characteristic, at least a first number ofthe sections form a color chart of a plurality of different colors andat least a second number of the sections are optical phantoms that matchrespective ones of a number of spectral characteristics of livingtissue.

The fiducial marker may further include a scattering layer overlying atleast a first set of the sections. The scattering layer may have anumber of characteristics that simulate a number of opticalcharacteristics of at least one layer of the living tissue. The opticalcharacteristics may be those of skin. The second number of the sectionsmay include at least one of a first section having a selective spectralabsorption at a waveband of about 330 nm to about 500 nm, a secondsection having a selective spectral absorption at a wavelength at about415 nm, about 515 nm, or about 590 nm, a third section having aselective spectral absorption at a waveband of about 340 nm to about 400nm, a fourth section having a selective spectral fluorescence at awaveband of about 450 nm to about 550 nm, or a fifth section having aselective spectral absorption at about 550 nm, about 630 nm, about 730nm, or about 820 nm or a reflection peak at about 514 nm, about 606 nmor about 739 nm. The fourth section may include at least one offluorescein or 5-carboxyfluorescein. The second number of the sectionsmay include each of a melanin layer phantom section having a selectivespectral absorption at a waveband of about 330 nm to about 500 nm, ahemoglobin layer phantom section having a selective spectral absorptionat a wavelength at approximately 415 nm, about 515 nm, or about 590 nm,a first collagen layer phantom section having a selective spectralabsorption at a waveband of about 340 nm to about 400 nm, a secondcollagen layer phantom section having a spectral fluorescence at awaveband of about 450 nm to about 550 nm, and a fifth section having aselective spectral absorption at about 550 nm, about 630 nm, about 730nm, or about 820 nm or a reflection peak at about 514 nm, about 606 nmor about 739 nm. At least a second set of the sections may not beoverlaid by the scattering layer. The colors in the color chart mayinclude at least one of black or white. The colors in the color chartmay include a plurality of different shades of grey. A first set of thesections may include a first color chart having a black section, a whitesection, a plurality of sections each of which is a respective shade ofgrey and a plurality of sections each of which is a respective one of aplurality of additional colors, all the sections of the first setoverlaid by a scattering layer, and a second set of the sectionsincludes a second color chart having a black section, a white section, aplurality of sections each of which is a respective shade of grey and aplurality of sections each of which is a respective one of a pluralityof additional colors, none of the sections of the second set overlaid bya scattering layer. The defined profile may be a polygon.

A system to image bodily tissues may be summarized as including aphysical fiducial marker selectively positionable at least proximate aregion of interest on a portion of a bodily tissue to be imaged, thephysical fiducial marker including a substrate having a defined profileand bearing a plurality of sections having respective wavelengthselective absorption, reflectance or florescence characteristic, atleast a first number of the sections form a color chart of a pluralityof different colors and at least a second number of the sections areoptical phantoms that match respective ones of a number of spectralcharacteristics of living tissue; at least one light source operable toproject a virtual fiducial marker at least proximate the region ofinterest on the portion of the bodily tissue to be imaged, the virtualfiducial marker having a defined profile and a plurality of definedshapes; and an image capture device having a field of view andconfigured to capture digital images of bodily tissue including theregion of interest, the physical fiducial marker and the virtualfiducial marker all encompassed by the field of view of the imagecapture device. The virtual fiducial marker may be projected with theplurality of defined shapes as straight line segments. The virtualfiducial marker may be projected with the profile of a circle and withthe plurality of defined shapes as straight line segments emanating froma center point of the circular profile. The defined profile of thephysical fiducial marker may be a polygon. The colors in the color chartmay include at least one of black, white, a plurality of differentshades of grey, a plurality of additional colors that are not black,white or grey.

A method of operating a system for use in tissue analysis may besummarized as including assessing by at least one processor of thesystem a change in at least one of a level of hydration, a level ofblood flow, a total number of wrinkles, a size of at least one wrinkle,a total number of blemishes, or a size of at least one blemish between afirst digital image of a region of interest of a bodily tissue and atleast one subsequent digital image of the region of interest of thebodily tissue, where the first digital image represents the region ofinterest prior to a first application of a cosmetic, a moisturizer, atherapeutic or a therapeutic treatment and the at least one subsequentdigital mage represents the region of interest after the firstapplication of the cosmetic, the moisturizer, the therapeutic or thetherapeutic treatment; and reporting by the at least one processor ofthe system the assessed difference in a visual form. Assessing a changein at least one of a level of hydration, a level of blood flow, a totalnumber of wrinkles, a size of at least one wrinkle, a total number ofblemishes, or a size of at least one blemish between a first digitalimage of a region of interest of a bodily tissue and at least onesubsequent digital image of the region of interest of the bodily tissuemay include assessing a number of spectral characteristics of the regionof interest in the first and the at least one subsequent digital image.Assessing a number of spectral characteristics of the region of interestin the first and the at least one subsequent digital image may includeassessing a spectral absorption, reflectance or fluorescence response ofa number of layers of skin characteristic of water, hemoglobin, andcollagen.

The method may further include comparing by the at least one processorof the system an appearance of at least one shape of at least a firstfiducial marker in a first digital image of a portion of a tissue to atleast one defined actual shape of the fiducial marker; comparing by theat least one processor of the system an appearance of each of aplurality of sections of the fiducial marker in the first digital imageto respective ones of defined sections of the fiducial marker includinga number of tissue phantoms each having a respective spectralcharacteristic that matches a respective spectral characteristic oftissue of a type represented in the first digital image; and at leastone of correlating, normalizing, or correcting at least the firstdigital image based at least in part on the comparisons. The assessingmay be performed after the at least one of correlating, normalizing, orcorrecting at least the first digital image based at least in part onthe comparisons.

A system for use in tissue analysis may be summarized as including atleast one processor; and at least one nontransitory storage medium thatstores processor executable instructions which when executed cause theat least one processor to: assess a change in at least one of a level ofhydration, a level of blood flow, a total number of wrinkles, a size ofat least one wrinkle, a total number of blemishes, or a size of at leastone blemish between a first digital image of a region of interest of abodily tissue and at least one subsequent digital image of the region ofinterest of the bodily tissue, where the first digital image representsthe region of interest prior to a first application of a cosmetic, amoisturizer, a therapeutic or a therapeutic treatment and the at leastone subsequent digital mage represents the region of interest after thefirst application of the cosmetic, the moisturizer, the therapeutic orthe therapeutic treatment; and report the assessed difference in avisual form. The instructions may cause the at least one processor toassess a change in at least one of a level of hydration, a level ofblood flow, a total number of wrinkles, a size of at least one wrinkle,a total number of blemishes, or a size of at least one blemish between afirst digital image of a region of interest of a bodily tissue and atleast one subsequent digital image of the region of interest of thebodily tissue by assessing a number of spectral characteristics of theregion of interest in the first and the at least one subsequent digitalimage. The instructions may cause the at least one processor to assess anumber of spectral characteristics of the region of interest in thefirst and the at least one subsequent digital image by determining aspectral absorption, reflectance or fluorescence response of a number oflayers of skin characteristic of water, hemoglobin, and collagen. Theinstructions may cause the at least one processor to: compare anappearance of at least one shape of at least a first fiducial marker ina first digital image of a portion of a tissue to at least one definedactual shape of the fiducial marker; compare an appearance of each of aplurality of sections of the fiducial marker in the first digital imageto respective ones of defined sections of the fiducial marker includinga number of tissue phantoms each having a respective spectralcharacteristic that matches a respective spectral characteristic oftissue of a type represented in the first digital image; and at leastone of correlate, normalize, or correct at least the first digital imagebased at least in part on the comparisons. The at least one processormay perform the assessing after performing the at least one ofcorrelation, normalization, or correction of at least the first digitalimage based at least in part on the comparisons. The at least oneprocessor performs may includes a therapy recommendation in the reportbased on the assessment.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

In the drawings, identical reference numbers identify similar elementsor acts. The sizes and relative positions of elements in the drawingsare not necessarily drawn to scale. For example, the shapes of variouselements and angles are not drawn to scale, and some of these elementsand angles are arbitrarily enlarged and positioned to improve drawinglegibility. Further, the particular shapes of the elements as drawn, arenot intended to convey any information regarding the actual shape of theparticular elements, and have been solely selected for ease ofrecognition in the drawings.

FIG. 1 is a schematic diagram that illustrates a tissue imaging systemand a subject tissue, showing an orientation of the imaging to thesubject tissue, according to one illustrated embodiment.

FIG. 2 is a schematic diagram that illustrates a field of view whichencompasses a subject tissue, including an area of interest and afiducial marker, showing a spatial context of the subject components andthe fiducial marker within the field of view or relative to the area ofinterest (x, y), according to one illustrated embodiment.

FIG. 3 is a schematic diagram of an tissue imaging system and tissueimage processing host computing system, remotely located from andcommunicatively coupled to the tissue imaging system, according to oneillustrated embodiment.

FIG. 4 is a graph that illustrates optical spectra of skin tissue withparticular consideration for optical layers that make up a basis forcomparative analysis, according to one illustrated embodiment.

FIG. 5 is a cross sectional view of a fiducial marker optical phantomwith a reflective layer overlying a number of reference sections of thetissue optical layers, according to one illustrated embodiment.

FIG. 6 is a top plan view of a fiducial marker, according to oneillustrated embodiment, which illustrates color sectors of the fiducialmarker; (a, b, c, d, e, f) being the primary and secondary referencecolors; (g, h) being the skin pigment reference colors; (i, j, k, l)being the black, white and grey scale; and (m, n, o, p) being theoptical phantom reference sections of the tissue optical layers.

FIG. 7 is a top plan view of a fiducial marker including a first portionhaving a first color chart and a scattering layer overlying the firstcolor chart, and a second portion having a second color chart which isnot overlaid by a scattering layer.

FIG. 8 is a top plan view of a physical and a virtual fiducial marker onat flat surface and one a surface that is not flat, according to oneillustrated embodiment, illustrating the change in geometry which isperceptible via variation in geometric elements or shapes of thefiducial marker.

FIG. 9 is a flow diagram illustrating an operation of a tissue imagingand digital image processing system, according to one illustratedembodiment.

DETAILED DESCRIPTION

In the following description, certain specific details are set forth inorder to provide a thorough understanding of various embodiments of theinvention. However, one skilled in the art will understand that theinvention may be practiced without these details. In other instances,well-known structures associated with cameras, imagers, scanners,optics, computers, computer networks, data structures, databases, andnetworks such as the Internet or cellular networks, have not beendescribed in detail to avoid unnecessarily obscuring the descriptions ofthe embodiments of the invention.

Unless the context requires otherwise, throughout the specification andclaims which follow, the word “comprise” and variations thereof, such as“comprises” and “comprising” are to be construed in an open, inclusivesense, that is as “including but not limited to.”

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure or characteristicdescribed in connection with the embodiment is included in at least oneembodiment of the present invention. Thus, the appearances of thephrases “in one embodiment” or “in an embodiment” in various placesthroughout this specification are not necessarily all referring to thesame embodiment. Furthermore, the particular features, structures, orcharacteristics may be combined in any suitable manner in one or moreembodiments.

The headings provided herein are for convenience only and do notinterpret the scope or meaning of the claimed invention.

As used herein and in the claims, “spectral effects” means theabsorption, reflection, exposure levels, white balance and fluorescenceand variations for optical conditions such as chromatic aberrations andfocus, and spherical aberrations and spatial corrections such as threedimensional characteristics and the orientation of the tissue imagingsystem in Cartesian space.

As used herein and in the claims, “complex interactions” means theinteraction of light in the various types of tissue due to tissuelayers, spectral effects where a subject image is transformed into acoordinate system and imaging conditions are the result of therelationship between various optical spectra and the tissue of interest.

As used herein and in the claims, “lesion shape” means the threedimensional shape, volume and/or depth of infiltration of a skin lesionincluding compensation for spectral effects and complex interactions,where digital images or digital photographs can be normalized using afiducial marker via computer image processing methods.

As used herein and in the claims, “analysis or diagnosis” means theresulting comparison of differences from one portion of an image toanother portion of the image, or from one image to another image. Astatic or timeline image sequence can be used by clinicians to evaluatethe significance of any changes due to the propagation and attenuationof light of certain defined wavelengths and for the different physicallayers of the skin, and data regarding the spectral distribution of thereflected and the back scattered light including compensation for lesionshape, spectral effects and complex interactions.

As used herein and in the claims, “fiducial marker” means a system forcorrecting for the variations in spectral power distribution from oneimage to another including a fiducial marker color chart with a physicalarrangement of known colors and used for color registration within theimage space including calibration of the reflected light from thesubject including comparing and adjusting with the color chart includingthe white balance and grey scale and where a tissue phantom on thefiducial marker is used for correlation of light from deeper tissue andto minimize the effect from surface reflection.

As used herein and in the claims, the term “about” refers to a +/−10%variation from the nominal value. It is to be understood that such avariation is always included in a given value provided herein, whetheror not it is specifically referred to.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this application relates.

Overview

FIG. 1 shows a tissue imaging system 100 according to one illustratedembodiment, which may be used to capture images of tissue 102 foranalysis and diagnosis.

The tissue imaging system 100 includes a digital camera 104 or otherimage capture device operable to capture images of tissue 102. Oneexample of a professional digital camera that may be suitable is an AltaU series digital camera commercially available from Apogee Instruments.Alternatively, a consumer style camera may be suitable, for example anSD 1300 digital camera commercially available from Canon.

The tissue imaging system 100 may include one or more excitation sources106, which may, or may not, be integral to the digital camera 104.Suitable excitation sources 106 may include a xenon flash tube or bulband associated circuitry. The tissue imaging system 100 may include oneor more excitation filters 108 positioned between the excitationsource(s) 106 and the tissue 102 to filter electromagnetic radiationemitted by the excitation source(s) 106.

The tissue imaging system 100 may include one or more imaging lenses110. The imaging lenses 110 may, or may not, be integral to the digitalcamera 104. The imaging lenses 110 may be used to adjust a focal pointand/or depth of field of the tissue imaging system 100. The tissueimaging system 100 may include one or more imaging filters 112positioned between the tissue 102 and the imaging lenses 110 or digitalcamera 104 to filter electromagnetic radiation returned (e.g.,reflected, emitted) by the tissue 102. Within the image field of view114 of the digital camera 104 is placed a fiducial marker 116 whichfacilitates normalization between digital images.

FIG. 2 shows the field of view 116 of the digital camera 104 (FIG. 1),according to one illustrated embodiment.

The field of view 114 encompasses a portion of tissue 102, whichincludes a region of interest 118. The region of interest 118 may takethe form of a lesion, growth or some other structure of, or on, thetissue 102. The field of view 114 also encompasses the fiducial marker116.

FIG. 3 shows a tissue imaging and digital image processing system 200according to one illustrated embodiment.

The tissue imaging and digital image processing system 200 includes oneor more tissue imaging systems 100, for example identical or similar tothe tissue imaging system 100 discussed in reference to FIGS. 1 and 2.The tissue imaging and digital image processing system 200 also includesone or more tissue image processing host computer systems 202. Thetissue imaging system(s) 100 is(are) communicatively coupled to thetissue image processing host computer system(s) 202 by one or morecommunications channels, for example the Internet 206, one or more localarea networks (LANs) 208 or wide area networks (WANs) 210. The tissueimage processing host computer system 202 will at times be referred toin the singular herein, but this is not intended to limit theembodiments to a single device since in typical embodiments, there maybe more than one clinic, hospital, or image processing service orfacility involved. Unless described otherwise, the construction andoperation of the various blocks shown in FIG. 2 are of conventionaldesign. As a result, such blocks need not be described in further detailherein, as they will be understood by those skilled in the relevant art.

The tissue image processing host computer system 202 may take the formof a conventional mainframe computer, mini-computer, workstationcomputer, personal computer (desktop or laptop), or handheld computer.Non-limiting examples of commercially available computer systemsinclude, but are not limited to, an 80×86 or Pentium seriesmicroprocessor from Intel Corporation, U.S.A., a PowerPC microprocessorfrom IBM, a Sparc microprocessor from Sun Microsystems, Inc., a PA-RISCseries microprocessor from Hewlett-Packard Company, or a 68xxx seriesmicroprocessor from Motorola Corporation.

The tissue image processing host computer system 202 may include one ormore processing units 212 a, 212 b (collectively 212), a system memory214 and a system bus 216 that couples various system componentsincluding the system memory 214 to the processing units 212. Theprocessing units 212 may be any logic processing unit, such as one ormore central processing units (CPUs) 212 a, digital signal processors(DSPs) 212 b, application-specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs), etc. The system bus 216 can employ anyknown bus structures or architectures, including a memory bus withmemory controller, a peripheral bus, and a local bus. The system memory214 includes read-only memory (“ROM”) 218 and random access memory(“RAM”) 220. A basic input/output system (“BIOS”) 222, which can formpart of the ROM 218, contains basic routines that help transferinformation between elements within the tissue image processing hostcomputer system 202, such as during start-up.

The tissue image processing host computer system 202 may include a harddisk drive 224 for reading from and writing to a hard disk 226, anoptical disk drive 228 for reading from and writing to removable opticaldisks 232, and/or a magnetic disk drive 230 for reading from and writingto magnetic disks 234. The optical disk 232 can be a CD-ROM, while themagnetic disk 234 can be a magnetic floppy disk or diskette. The harddisk drive 224, optical disk drive 228 and magnetic disk drive 230 maycommunicate with the processing unit 212 via the system bus 216. Thehard disk drive 224, optical disk drive 228 and magnetic disk drive 230may include interfaces or controllers (not shown) coupled between suchdrives and the system bus 216, as is known by those skilled in therelevant art. The drives 224, 228 and 230, and their associatedcomputer-readable storage media 226, 232, 234, may provide nonvolatileand non-transitory storage of computer readable instructions, datastructures, program modules and other data for the tissue imageprocessing host computer system 202. Although the depicted tissue imageprocessing host computer system 202 is illustrated employing a hard disk224, optical disk 228 and magnetic disk 230, those skilled in therelevant art will appreciate that other types of computer-readablestorage media that can store data accessible by a computer may beemployed, such as magnetic cassettes, flash memory, digital video disks(“DVD”), Bernoulli cartridges, RAMs, ROMs, smart cards, etc.

Program modules can be stored in the system memory 214, such as anoperating system 236, one or more application programs 238, otherprograms or modules 240 and program data 242. Application programs 238may include instructions that cause the processor(s) 212 toautomatically normalize digital images or information therefrom based onfiducial markers in those digital images and/or compare tissue orlesions between normalized digital images. Other program modules 240 mayinclude instructions for handling security such as password or otheraccess protection and communications encryption. The system memory 214may also include communications programs for example a Web client orbrowser 244 for permitting the tissue image processing host computersystem 202 to access and exchange data with sources such as Web sites ofthe Internet, corporate intranets, extranets, or other networks asdescribed below, as well as other server applications on servercomputing systems such as those discussed further herein. The browser244 in the depicted embodiment is markup language based, such asHypertext Markup Language (HTML), Extensible Markup Language (XML) orWireless Markup Language (WML), and operates with markup languages thatuse syntactically delimited characters added to the data of a documentto represent the structure of the document. A number of Web clients orbrowsers are commercially available such as those from Mozilla, Googleand Microsoft of Redmond, Wash.

While shown in FIG. 2 as being stored in the system memory 214, theoperating system 236, application programs 238, other programs/modules240, program data 242 and browser 244 can be stored on the hard disk 226of the hard disk drive 224, the optical disk 232 of the optical diskdrive 228 and/or the magnetic disk 234 of the magnetic disk drive 230.

An operator can enter commands and information into the tissue imageprocessing host computer system 202 through input devices such as atouch screen or keyboard 246 and/or a pointing device such as a mouse248, and/or via a graphical user interface. Other input devices caninclude a microphone, joystick, game pad, tablet, scanner, etc. Theseand other input devices are connected to one or more of the processingunits 212 through an interface 250 such as a serial port interface thatcouples to the system bus 216, although other interfaces such as aparallel port, a game port or a wireless interface or a universal serialbus (“USB”) can be used. A monitor 252 or other display device iscoupled to the system bus 216 via a video interface 254, such as a videoadapter. The host computer system tissue image processing can includeother output devices, such as speakers, printers, etc.

The tissue image processing host computer system 202 can operate in anetworked environment using logical connections to one or more remotecomputers and/or devices. For example, the tissue image processing hostcomputer system 202 can operate in a networked environment using logicalconnections to one or more network server computer systems (not shown).Communications may be via a wired and/or wireless network architecture,for instance wired and wireless enterprise-wide computer networks,intranets, extranets, and the Internet. Other embodiments may includeother types of communication networks including telecommunicationsnetworks, cellular networks, paging networks, and other mobile networks.

As explained herein, the tissue image processing host computer systems202 may perform human tissue image correlation, image analysis,normalization and/or correction of optical exposure, spectral andspatial distribution in order to compensate for the surface reflections,sub surface tissue interactions and spatial orientation of theexcitation and imaging axes to the subject tissue.

The tissue imaging and tissue image processing system 200 should beflexible such that clinicians can model tissue image data in differentforms in order to cross reference and compare data from various spectralcomponents and/or from different digital images. The ability for humaninterpretation between images is substantial and the tissue imaging andtissue image processing system 202 may enable variations to be seenbetween images, even when the spectral and spatial optical conditions orthe image resolution or sensitivity are compromised or vary betweentimes of image capture.

The use of the tissue image processing host computer system 202 tocorrect an image involves more than adjusting the same exposure levelsand white balance. The introduction of a color card or fiducial markerinto the field of view of the digital camera or image can enable thetissue image processing host computer systems 202 to automatically makecorrections corresponding to the spectral effects in a tissue sample dueto the complex interactions of light absorption, reflection andfluorescence and to automatically make spatial corrections to correcterrors or differences due to the three dimensional characteristics andthe orientation of the tissue imaging system 100 in Cartesian space.

The three dimensional shape, volume and depth of infiltration of a skinlesion are significant factors for ongoing clinical analysis. The tissueimage processing host computer system 202 (FIG. 3) uses computer imageprocessing methods on digital images or digital photographs to normalizemuch of the information, and to enhance critical features such as LesionShape and lesion borders and metabolically active areas that can aid inclinical analysis and diagnosis. The tissue image processing hostcomputer 202 may produce appropriate digital or physical reports 270.

While the tissue imaging system 100 is described in terms of a digitalcamera and an excitation source 106 such as a light source or Xenonflash tube, other types of sensors such as spectrometers and other typesof modulated excitation (e.g., light) sources could be employed. Thetissue image processing host computer system 202 executes one or morecomputer software programs, processes or algorithms to perform variousimage processing techniques on the optical spectral data from thecaptured digital image. The host computer system is programmed to createa virtual three dimensional reconstruction or three dimensional digitalmodel of the lesion, and to add multispectral data and image timelinedata to the digital model or to a data structure associated with thedigital model. The resulting digital model may be displayed as images onan appropriate device (e.g., LCD display, cathode ray tube display).Analysis of differences from normal versus lesion data in a static ortimeline image sequence can be used by clinicians to evaluate thesignificance of any changes. The tissue image processing host computersystem 202 may also be programmed to establish a baseline of what isnormal for a given patient from an optical/molecular perspective, andthis data used to contribute to the normalization of digital images orbe used in a timeline as at least part of a diagnostic test.

FIG. 4 shows a graph 400 that illustrates optical spectra of skin tissuewith particular consideration for optical layers that make up a basisfor comparative analysis, according to one illustrated embodiment.

Characterizing the Optical Inhomogeneity

The tissue image processing host computer system 202 is programmed tocorrect for electromagnetic radiation returned from the tissue 102 (FIG.1). For example, the tissue image processing host computer system 202may correct the reflected light spectra, in the visible portion of theelectromagnetic spectrum, of the skin area surrounding a skin lesion.Electromagnetic radiation (e.g., light in the visible and non-visibleportions the electromagnetic spectrum) undergoes absorption and multiplescattering in the skin and is back scattered and re-emerges carryinginformation which characterizes or is indicative of certain physicalcharacteristics of the structure of the skin and/or the lesion. Thepropagation and attenuation of electromagnetic radiation of certainwavelengths (e.g., light) varies for the different layers of the skin.The tissue imaging and digital image processing system 200 captures dataregarding the spectral distribution of the electromagnetic radiation(e.g., light) which is reflected and/or back scattered and/or fluorescesfrom the tissue (e.g., skin, lesion). The tissue image processing hostcomputer system 202 can separate the wavelengths or wavebands in thedigital image, and compare the ratios of the respective radiant spectralintensity of the wavelengths or wavebands. This allows for evaluation ofspecific areas within a single image. This also allows for directcomparison at a later date to the data obtained from other images of thesame tissue 102 (FIG. 1) or region of interest 118 (FIG. 2). In thisway, a spectral distribution of certain areas of the digital image canbe used to manage or aid in the interpretation of a digital image ortimeline (i.e., sequence) of digital images. Further, the data from thedigital image or sequence of digital images can be used to aid in theinterpretation of the molecular structure of the subject tissue.

In use, the tissue imaging system 100 (FIG. 1) may capture a firstsubject digital image at a first time, and may capture further digitalimages at later times. Thus, the captured digital images may represent asampling of the same scene or object or area at different times, and/orunder different optical conditions. The tissue image processing hostcomputer system 202 transforms the first subject digital image I₁ into adefined coordinate system and certain conditions are highlighted withinan image layer L_(n) or as metadata. One layer might represent therelationship between various optical spectra. As further or sequentialdigital images are added I_(1+n), tissue image processing host computersystem 202 can compare those digital images on a layer to layerL_(1 . . . n) basis by registering the new digital image in spatialcoordinates and optical relationships. Deformation of the subjectdigital image due to either poor imaging technique, or morphologicalchange is made to evaluate optical changes at specific coordinates inthe digital image within the layers L_(n) or compare the changes to thevariations from the first subject digital image I₁ or other layers indifferent digital images along a timeline.I ₁ →L _(1 . . . n)I ₂ →L _(1 . . . n)

In this manner, the tissue image processing host computer system 202compares the layers of each digital image such as the difference of onelayer to the average:(I ₁ L ₁ +I ₂ L ₂)/2−I ₁ L ₁

In which case the tissue image processing host computer system 202 mayrecalculate the individual pixel values. In certain cases only the pixelvalues of specific x, y coordinates would be used.

One layer might represent the relationship between various opticalspectra. Optical changes such as spectral changes or optical density atspecific coordinates in the digital image O_((xn, yn)) can be referencedto compare the changes to the variations from another subject digitalimage I_(1+n), such as the coordinates that were indicative of thelesion area. Alternatively, or additionally, a numerical value could becreated by a clinician identifying a cross sectional area where ahistogram (hist) would be generated to visualize the probability densityof certain optical spectra within, or between, layers such as for theset of specific x, y values in chosen layers:hist{I ₁ L ₁(x _(n) ,y _(n))}

Histograms or other types of analysis such as derivatives or ratios canbe used to determine the changes between areas of interest 118 (FIG. 1)within a layer or between layers.

Fiducial Marker

One or more fiducial markers 116 (FIG. 1) may be employed to provideinformation in the digital images which allows the tissue imageprocessing host computer system 202 to perform various image processingacts (e.g., normalization, comparison, three dimensional modeling). Asexplained in detail below, the fiducial markers 116 may take the form ofphysical objects (e.g., physical media) which are placed at, on, orproximate the region of interest 118 (FIG. 1) of the tissue 102 forimage capture Alternatively, or additionally, the fiducial markers 116may be virtual, taking the form of structure light projected toilluminate an area at, on, or proximate the region of interest 118(FIG. 1) of the tissue 102 for image capture. Whether physical fiducialmarkers, virtual fiducial markers or both are employed, a knowledge ofbaseline or starting physical characteristics of the fiducial markers116 is used to identify differences (e.g., spectral, geometric ortopological) between the baseline or starting physical characteristicsand how the fiducial markers 116 appear in a captured digital image.Such allows characterization of differences between captured images,which differences may arise due to system variations or ambientenvironment variations, which variations are unrelated to variations inthe tissue itself.

As best illustrated in FIG. 6, the fiducial markers 116 have a definedshape or two dimensional profile. While illustrated as rectangular,other shapes may be employed, for example hexagonal, octagonal, oranother polygon. In some instances, fiducial markers may have othernon-polygonal profiles, for example oval or circular, although such maybe less preferable since such non-polygonal shapes may limit the amountof information regarding orientation which can otherwise be discernedfrom the appearance of the fiducial marker 116 in a digital image. Insuch instances it may be advantageous to include additional definedshapes in the fiducial markers 116, for radial line segments.

Also as best illustrated in FIG. 6, the fiducial markers 116 may takethe form of a plurality of discrete portions, denominated as sectionsherein. Each portion or section may have a respective characterizingphysical property, for example a respective spectral electromagneticradiation absorption or reflectance property or characteristic. Thus,for example, the fiducial marker 116 may appear as an array of differentcolor, grey scale or white sections, each section preferentiallyabsorbing certain wavelengths or bands of wavelengths while reflecting,back scattering or fluorescing other wavelengths or bands ofwavelengths. While the sections may typically arranged in a twodimensional array, such as illustrated in FIG. 6, some fiducial markersmay take the form of a one dimensional or linear array, while otherfiducial markers may include an unordered array or collection ofsections. The characterizing physical properties (e.g., spectralabsorption and/or reflectance properties) and relative positions of eachsections are known to the tissue image processing host computer system202, at least during image processing, as is the baseline or startingshape or profile.

These sections may be organized by function or characterizing property.For example, a number of sections may form a color chart, includingprimary/secondary color portion (sections labeled A-F), skin pigmentreference colors portion (sections labeled G-H), black, white, greyscale portion (sections labeled I-L). A number of sections may form atissue phantom portion (sections labeled M-P). Each portion mayrespectively include one or more distinct sections with respectivephysical characteristics such as selective wavelength absorption,reflectance and/or florescence.

For example, the primary/secondary color portion (sections labeled A-F)of the color chart may take the form of a physical arrangement ofdefined primary and/or secondary colors, which can be used for colorregistration within the image space. For instance, sections A-F mayappear as blue, teal, green, magenta, red, and yellow, respectively. Thegrey scale portion of the color chart may be an arrangement of sectionsused as a middle gray reference of in the order of 13-18% reflectance,and the black or white balance portion of the color chart may be in theorder of 90% reflectance and used by the tissue image processing hostcomputer system 202 to compensate for variations in apparent opticalexcitation such as variations in excitation source to subject distance(i.e., distance between excitation source 106 and target tissue 102).For instance, sections I-L may appear as dark grey, white, black, lightgrey, respectively. For instance, the sections G and H may be brown anda Caucasian skin tone (e.g., tan), respectively. For instance, thesections M-P may appear as violet, blue, purple and light blue,respectively.

Corrections are to be used for calibration of the light returned (e.g.,reflected, back scattered, fluoresced) from the subject tissue 102. Thetissue image processing host computer system 202 can correct multipledigital images by comparing and adjusting for the appearance of theprimary/secondary color portion (sections labeled A-F), black, whitebalance and grey scale portion (sections labeled I-L) in the digitalimage based on the known values of the various sections. In more complexoptical correlation between digital images, tissue phantom portion(sections labeled M-P) on the fiducial marker 116 may be used forcorrelation of light from deeper tissue and to minimize the effect fromsurface reflection. Thus, the tissue image processing host computersystem 202 uses the appearance of the fiducial marker 116 in digitalimages to measure and correct for variations in spectral powerdistribution from one digital image to another digital image.

As best illustrated in FIG. 5, a tissue phantom fiducial marker 116 is astructure to correct for the reflection and the backscatter of thedominant wavebands of the skin. The tissue phantom portion of thefiducial marker 116 includes a wavelength selective portion 502 that hasan overlying surface or coating 504. The overlying surface or coating504 causes some surface scatter while also allowing transmission of somelight, and has a known optical density O_(n). The wavelength selectiveportion 502 of the tissue phantom portion may include a plurality ofsections or swatches of different colors (five illustrated in FIG. 5)502 a-502 e that advantageously represent the spectral absorption andreflection of collagen and also of hemoglobin. The overlying surface orcoating 504 may advantageously take the form of a matt surface thatyields a five to seven percent reflection.

FIG. 7 shows a physical fiducial marker 700, according to oneillustrated embodiment.

The physical fiducial marker 700 has a first portion 702 a whichincludes a scatter layer 702 a overlying a first plurality of sectionsA-P. The sections A-P of the first plurality each have respectivedifferent colors or wavelength selective spectral absorption,reflectance, and/or florescence characteristics (sixteen illustrated,labeled A-P), identical or similar to those discussed above. Thephysical fiducial marker 700 has at least a second portion 702 b whichomits the scatter layer overlying a second plurality of sections. Thesections A-P of the second plurality each have respective differentcolors or wavelength selective spectral absorption, reflectance, and/orflorescence characteristics (sixteen illustrated, labeled A-P),identical or similar to those discussed above.

Each portion 702 a, 702 b may include the same set and spatialarrangement of sections or colors. For example, each sector A-P of thefirst portion 702 a may be a respective one of 16 colors, while eachsector A-P of the second portion 702 b may be a respective one of thesame 16 colors. The colors of the sectors A-P of the second portion 702b may be spatially arranged in the same order or relative positions withrespect to one another as the order or relative positions of the sectorsA-P of the first portion 702 a. Thus, sector A of both the first portion702 a and the second portions 702 b may both be, for example red orotherwise have the same wavelength selective spectral absorption,reflectance and/or florescence characteristics.

The inclusion of a scatter layer 504 (FIG. 5) on the first portion 702 aand omission of such from the second portion 702 b facilitates automatednormalization that corrects for spectral distribution of a sensor (e.g.,image sensor such as an array of charge coupled device, or CMOS imagesensor). Notably, sensors may respond differently in changing ambientconditions, such as changing light and/or temperature conditions. Thecorrection for sensitivity across the spectral distribution and scatterat specific wavebands can be used to optimize the image processing,especially to compare the ratios of sectors.

The fiducial marker 116 (FIGS. 1 and 2) may be virtual, being formed byprojecting structured light from the excitation source 106 (FIG. 1)which forms a pattern of light on the subject tissue 102. The projectionmay be of multi-dimensions and include variations of optical spectra.The projection may be sweeping by use of a laser scanner, holographicscanner or monochromator or may achieved via one or more filters ordiffractive optical lenses to fit over the excitation source 106(FIG. 1) for example a flash tube or bulb of a digital camera 104. Usingstructured light allows the tissue image processing host computer system202 to perform automated tissue layer analysis due to the variations intriangulation with multiple spectra. Variations in optical spectra canbe used to measure or otherwise determine the molecular opticalcharacter of the subject without interference from out of bandwavelengths. The sensor or digital camera 104 captures the incidentlight from the surface of the subject tissue 102, which provides adigital image containing information which can be analyzed and/orprocessed to create an image map.

FIG. 8 shows a physical fiducial marker 802 a, 802 b (collectively 802)and projected or virtual fiducial marker 804 a 804 b (collectively 804)used in combination.

The physical fiducial marker 802 a, 802 b has a defined shape orprofile, rectangular in FIG. 8, and includes a plurality of sectors,each with respective spectral absorption, reflectance or florescencecharacteristics, as discussed above. When projected onto a flat surface,the projected or virtual fiducial marker 804 a has a defined shape. Forexample when projected onto a flat surface, as illustrated in the lowerportion of FIG. 8, the projected or virtual fiducial marker 804 a mayhave a circular profile and a plurality of straight radial line segmentswhich emanate from a center point of the circular profile. However, whenprojected onto a surface that is not flat (e.g., lesion), the projectedor virtual fiducial marker 804 b has a shape that conforms to thenon-flat surface. For example when projected onto a lesion, asillustrated in the upper portion of FIG. 8, the projected or virtualfiducial marker 804 b the profile may be changed and/or the radial linesegments may no longer be straight but rather reflect the threedimensional contours of the lesion.

Thus, the physical fiducial marker 802 and projected or virtual fiducialmarker 804 can be used in combination to assure that the benefits ofeach are utilized. The physical fiducial marker 802 may advantageouslyprovide improved normalization and correction for spectral character,while the projected or virtual fiducial marker 804 may advantageouslyprovide improved shape correlation to spectral abnormalities.

The projected fiducial marker 804 can be of a form that providesinformation which allows the tissue image processing host computersystem 202 to effectively analyze three dimensional tissue (i.e., tissuethat has a relatively large change in curvature or change along theZ-axis over the XY area of tissue being imaged), such as the cervix. Ascatter distribution can be obtained in combination with shapemeasurement. The tissue image processing host computer system 202 cancompare this information at various wavelengths to create a spatial andspectral map of the tissue and the optical characteristics of thetissue.

As noted above, the projected or virtual fiducial marker 804 can be usedas various shapes to enable the collection of both spectral and spatialinformation. The projected or virtual fiducial marker can use opticallydiscrete parameters, for example projected lines, so as to measure thedistortion of the area of interest. The reflective nature of the subjecttissue can be measured or otherwise determined or assessed as localdistortion of the projected line such as optical saturation of thesensor versus incident reflection. In this respect it is noted that twospatially identical optical line projections at different wavelengthsdisplay different scatter characteristics. The tissue image processinghost computer system 202 can measure or otherwise determine such forvarious wavelengths of light to get a better measure of the reflection,absorption, transmission and fluorescence at coordinates within thedigital image. For example, two spatially identical projections of aline, varied only in their wavelength, might have greater reflectance atone point at a specific wavelength versus another. This enables the postprocessing of digital images to account for reflectance artifacts.

FIG. 9 is a flow diagram illustrating a method 900 of operating a tissueimaging and digital image processing system 202, according to oneillustrated embodiment. The method 900 is exemplary. In use, the method900 may include additional acts, omit some acts, and/or perform acts indifferent orders. The method 900 is presented as an overview. Many ofthe specifics of performing the various acts of the method 900 aredescribed in detail herein.

At 902, the tissue image processing host computer system 202 performsspatial correction on a digital image. The tissue image processing hostcomputer system 202 may rely on the difference between how the fiducialmarker appears in the digital image and a known appearance of thefiducial marker. Spatial correction may, for example correct for variousdigital imaging system misalignments and is generally discussedelsewhere herein.

At 904, the tissue image processing host computer system 202 registersand/or rectifies the digital image. Image registration and/orrectification are discussed in more detail elsewhere herein.

At 906, the tissue image processing host computer system 202 maygenerate a region of interest spatial baseline (e.g., lesion spatialbaseline). The spatial baseline may facilitate comparisons over time.Spatial baseline generation is discussed in more detail elsewhereherein.

At 908, the tissue image processing host computer system 202 maygenerate a three dimensional model of the region of interest (e.g.,lesion). The tissue image processing host computer system 202 may employinformation stored in a database of historical parameters 910, stored onone or more nontransitory computer readable storage mediums. The threedimensional model facilitates comparisons, and is discussed in moredetail elsewhere herein.

At 912, the tissue image processing host computer system 202 may performspectral correction on the digital image. The tissue image processinghost computer system 202 may rely on the difference between how thefiducial marker appears in the digital image and a known appearance ofthe fiducial marker. Spectral correction may, for example, correct forvarious differences in imaging conditions, for example variations inlighting, and is generally discussed elsewhere herein.

At 914, the tissue image processing host computer system 202 may performspectral normalization on the digital image. The tissue image processinghost computer system 202 may employ information stored in the databaseof historical parameters 910 to perform spectral normalization. Spectralnormalization is discussed in detail elsewhere herein.

At 916, the tissue image processing host computer system 202 may createspectral layers in the digital image file, storing spectral informationthereto Layers of the digital image file are discussed in detailelsewhere herein.

At 918, the tissue image processing host computer system 202 determinesreflection, absorption, fluorescence values or characteristics for thedigital image. As described elsewhere herein, the tissue represented inthe image may be characterized by is spectral characteristics, inparticular in the particular wavelengths of wave bands which the tissue,or portions thereof, absorb, reflect or fluoresce. Determination of thereflection, absorption, fluorescence values or characteristics aredescribed in detail elsewhere herein.

At 920, the tissue image processing host computer system 202 generates aregion of interest spectral baseline (e.g., lesion spectral baseline).The spectral baseline allows spectral changes in the region of interestto be easily and accurately compared and identified. Generation ofspectral baselines are discussed in detail elsewhere herein.

At 922, the tissue image processing host computer system 202 performspost processing. There are numerous possible post processing procedures,which are described elsewhere herein.

At 924, the tissue image processing host computer system 202 determinesspectral ratios. As described herein, spectral ratios may beparticularly advantageous for allow comparisons within a digital imageor between digital images. The determination and use of spectral ratiosare described in detail elsewhere herein.

At 926, the tissue image processing host computer system 202 determinesprobability indices. The tissue image processing host computer system202 may employ the lesion spectral baseline generated at 920 indetermining the probability indices. Probability indices facilitate thediagnosis of tissue, such as lesions, and the generation and use of suchare discussed in detail elsewhere herein.

At 928, the tissue image processing host computer system 202 generates athree dimensional digital model of the region of interest (e.g., lesion)incorporating the spectral changes. The generation of the threedimensional digital model and the benefits thereof are discussed indetail elsewhere herein.

At 930, the tissue image processing host computer system 202 mayregister the three dimensional digital model generated at 928 with athree dimensional map of a human body or portion thereof. The tissueimage processing host computer system 202 may employ information fromthe database of historical parameters generated at 910. The registrationmay facilitate analysis and/or diagnosis, as discussed in more detailelsewhere herein.

Normalization

In some embodiments, the tissue image processing host computer system202 may normalize an image time series (i.e., temporal sequence ofdigital images). Those digital images may have been captured over arelatively long time (e.g., decades, years, months) at any variety offrequencies or intervals, and/or over a relatively short time (e.g.,weeks, days, hours) at any variety of frequencies or intervals. In thecase of multiple digital images taken over variable ambient conditions,any difference of the spectral distribution between the reflectivecharacter of a digital image, such as those taken in low ambient lightversus digital images created with electronic flash, will reduce theprobability of confidence in an image correlation. Such a case willrequire numerical methods to correct or normalize the digital images.

The monotonicity of certain spectral relationships may be outside orapproaching the limit of the normal spectral distribution. This might betypical for the case where a digital image is normalized to the fiducialmarker 116, yet still requires to be normalized based on the spectralmarkers such as hemoglobin and collagen.

The presence by measure of reflective, absorptive, transmissive orfluorescent light, or relationship of one spectra to another iscomputationally bounded within certain limits of what is normal. Normalmay be determined by the spectral distribution ratios of healthy tissuein the person of interest or in comparison to a population. Once therehas been a normalization, certain optical relationships can be analyzedsuch as a probability distribution. One example of this is the opticaldensity compared to the percentage of the optical spectra that can beattributed to that of collagen. Another example is to remove the spectrathat are attributed to the surface skin and allow for sub surfaceanalysis.

In the case where spectral distribution is corrected by numericalmethods, a computed distribution that results in increases of a wavebandthat might normally cause fluorescence will not be able to assignfluorescent values outside of what is considered normal. However, byanalysis the distribution can be automatically assessed by the tissueimage processing host computer system 202 to determine if there arecorresponding increases in spectra that would relate to absorption andfluorescence. To correct for fluorescence in a time series ofnumerically processed digital images requires then that the digitalimages are assessed or analyzed within a probability index.

Some embodiments may advantageously employ digital images that aredisplayed in layers assigned to the wavebands of excitation and with aprobability index that allows certain images to be weighted in theirdiagnostic value.

The monotonicity of spectral changes, whether individual spectra orcomparative changes between spectra may show a trend; for instance, atrend that highlights a decreasing amount of collagen in one tissueversus another tissue. The tissue image processing host computer system202 may consider or assess a linearity of the function versus thenormalization.

Exposure and Color Correction

Exposure may vary with both the angle of incidence, the relative angleof the illumination (e.g. flash) to the subject tissue 102 (FIG. 1), theirradiance of the illumination and the spectral distribution of theillumination, the orientation of the subject tissue 102 from one digitalimage to another digital image with respect to the optical axes, thedistance and angle of the imaging optics to the subject tissue 102, anyfilters 108, 112 in the optical paths (arrows in FIG. 1) and shutterspeed, aperture and sensitivity of the imager or image capture device104.

In at least some embodiments, the tissue image processing host computersystem 202 is designed to compensate for the optical effects that varywith the optical exposure. Exposure may be initially established bycomparing the features of the fiducial markers 116 and correction forspectral distribution of the excitation source 106 (FIG. 1) and then thetissue characteristics. Compensation for reflection involves separationof the surface effects from the subsurface effects.

In at least some embodiments, color correction includes exposureanalysis based on the fiducial markers 116, modeling of the tissue depthprofile, and further includes use of information of the ratios of lesioncolors, width, fluorescence and reflection to that of surrounding tissuein a two dimensional normalization.

In at least some embodiments, color correction includes exposureanalysis based on the fiducial markers 116, modeling of the tissue depthprofile with greater degrees of specificity including spectralcorrection of non-lesion tissue components such as optical biomarkersfrom subsurface tissue excitation including, blood, oxygen, glucose,collagen, flavins, elastin, tryptophan, NADH etc.

Notably, the apparent optical exposure may also vary due to changes inthe subject, including hydration, blood flow, temperature and the ratiosof the natural skin components including epidermis, melanin, hemoglobin,collagen, bilirubin and other chromophores such as carotenoids andporphyrins. Further the apparent optical exposure may be altered by theuse of topical creams, cosmetics or by drug and food interactions withthe natural skin components. In such situations, the tissue imageprocessing host computer system 202 then adjust the digital image to abaseline normal of the reflected light, and remove any artifacts, andthe ratio of subsurface backscatter from one digital image to anotherdigital image can then be more easily compared by using a subsurfaceratio.

In at least some embodiments, the tissue image processing host computersystem 202 uses the ratios of the optical scattering of hemoglobin tothat of collagen as a reference to adjust and normalize the opticalexposure of the subsurface.

Optical Ratios

The reflective properties of the epidermis of one person versus anotherperson can vary significantly as so can the ratios of hemoglobin,collagen, melanin to epidermis from person to person. As a result, thecomplexity of human skin requires that in order to accurately referenceone digital image to another digital image, that an accurate model isrepresentative of each person. A system of optical layers for each imageI₁L₁ that conforms to the principle optical absorption bands is used toseparate the optical spectra as further described below.

An Epidermal Layer where scattering dominates absorption in the visiblespectrum of 500 nm to 600 nm I₁L_(E).

A Melanin Layer where the melanin absorption in the spectrum of 300 nmto 500 nm I₁L_(M) may be characterized by the optical density O orcomputed by comparison to other spectra from various molecular opticalsources at different wavelength Oλ. Melanin does not have a defined peakin the visible but its absorption coefficient decreases with the longerwavelengths.

A Hemoglobin Layer where the hemoglobin absorption in the visiblespectrum primary absorption of oxyhemoglobin peaks at 415 nm I₁L_(H).Adjustments to the hemoglobin layer may be made by comparison todeoxyhemoglobin such as primary absorption peaks at 430 nm or by usingthe absorption of hemoglobin to correct for artifacts in other layers,such as secondary absorption peaks for oxyhemoglobin at 542 nm and 577nm, and secondary absorption peaks for deoxyhemoglobin at 555 nm.

A Collagen Layer where the collagen absorption is measured in thevisible portion of the electromagnetic spectrum in the near UV such asthe optical region between 340 nm to 400 nm I₁L_(C). In addition, afluorescence peak would be measured in comparison to the absorption inthe optical wavelengths between 450 nm to 550 nm I₁L_(CF).

A Water Layer where the water absorption is measured in the visible andNIR spectrum at peaks 730 nm, 820 nm I₁L_(W).

In at least some embodiments, the tissue image processing host computersystem 202 uses the ratios of the optical scattering of the epidermis tothat of melanin as a reference to adjust and normalize the exposure ofthe subsurface.

The tissue image processing host computer system 202 can use anormalized ratio of one known spectral property to another to create apersonal optical profile. These spectral ratios or other ratios can beused as a personal optical profile for each person.

Morphology Correction

Correction for correlation of digital images should also include colorcorrection information in order to establish the border parameters. Thetissue borders are often areas where lesions can be analyzed for changesin tissue that might include fluorescence and reflection changes. Hencethe control of light exposure will have an impact on the measurement ofborders and boundaries which are used in physical size and growthcomparisons.

The analysis of boarders then can be made by measuring or otherwisedetermining the subsurface components of the optical spectral componentsof the tissue. The amount of collagen in tissue decreases as tissuebecomes neoplastic. Other components such as NADH increases and changesin blood flow are known to be synonymous with lesions and serve asexcellent markers for lesion boarders.

Lesion boarders can also be established by comparison of the surfaceoptical spectra to the subsurface spectra. This further allows for areference to be used in a timeline. A numerical index based on theoptical characteristics can be established.

In a multiple image series, normalizing the surface reflection in eachdigital image would be achieved by normalizing the color markers. Suchallows accurate comparison of the Epidermal Layer in the visiblespectrum of 500 nm to 600 nm I₁L_(E) to the Melanin Layer in thespectrum of 300 nm to 500 nm I₁L_(M) and comparing and adjusting ratiosas required at proximity to lesion site. Subsurface scatter may beadvantageously normalized via measurement or determination of ratiossuch as those of hemoglobin I collagen, a relative Hemoglobin Layerwhere absorption of oxyhemoglobin peaks at 415 nm I₁L_(H) to theCollagen Layer where absorption is between 340 nm to 400 nm I₁L_(C) or afluorescence peak in the optical wavelengths between 450 nm to 550 nmI₁L_(CF) and comparison and adjustment of ratios as required atproximity to lesion site border.

Once the borders spectral distribution has been established, theborders' spectra can be compared to the surrounding tissue and a lesionmap can be generated by visual analysis or by automated image processingmethods or techniques that track the pixel characteristics in thedigital image. The lesion map may be multidimensional and have referencelayers such as surface, sub-surface and other layers or depthcharacteristics as may be determined from the spectral analysis, such asareas of molecular activity, blood flow or tissue density.

In some cases tissue image processing host computer system 202 maycombine the optical data with spatial data to create true threedimensional digital models of the lesions or combined with physicalanatomical models, for instance in a form of rubber sheeting to createpseudo three dimensional physical models.

Growth comparisons are the combination of changes in color and changesin shape. The color data is either the absolute changes in total coloror the variation in the layers. Either or both can be used to monitorchange at any x, y coordinates, or as a method to reference the changesin a sample such as a cross section and its normalized spectraldistribution.

Combined Methods of Rectification.

The shape of the fiducial marker 116 (FIGS. 1 and 2) allows for imagerectification, but the lesion or some area or region of interest 118 inthe tissue 102 must be used to relate images to each other. In somecases, this can be done manually with obvious image characteristics. Inother cases, it can be done automatically (i.e., computationally by aprocessor such as a digital microprocessor) by examining areas withinthe image layers that have a notable and repeatable spatial and spectralvariation such as the difference between image layer coordinates in atime series:I _(2 . . . n) L _(C)(x _(n) ,y _(n))/I ₁ L _(C)(x _(n) ,y _(n))

Relative spherical and chromatic aberrations can be caused by theoptical system in normal function, or by variance to the conditions orsettings used such as excitation irradiance, focal length and aperture.

In at least some embodiments, the tissue image processing host computersystem 202 performs digital image rectification which converts digitalimages to a standard coordinate system for registration and astandardized method of optical correlation. This is done by matchingareas of the tissue 102 (FIGS. 1 and 2) or the fiducial marker 116 inthe source image I₁(x_(n),y_(n)) with areas of the tissue 102 or thefiducial marker 116 within the time series images I_(1 . . . n)(x_(n),y_(n)). This process is designed to overcome difficulties in clinicalimaging where accuracy or aberrations in area analysis cannot be welldefined, or where the digital images or layers lack clearly identifiablepoints with which to correlate between the digital images. A timesequence of digital images can also be used to correct for distortionsuch as variations of optical aberrations in the tissue imaging system100.

The shape of the tissue 102 may be rectified with a 3D topographic mapof the body or simulation thereof to compensate for distortion from thetissue topography at the area or region of interest 118. The geometriccorrection of digital images requires calculating the distortion at eachpixel or area, and then comparing the digital image to the properlocation in the 3D topographical map. The digital image is registeredwhen each pixel or area is placed in the correct precise 3D position orlocation. The adjustment may also take into consideration the excitationsource 106 (FIG. 1) and sensor 104 orientations and locations. Thismethod combines 3D model probabilities with measurements from digitalimages to provide precise, orthographically correct coordinatelocations. This process registers digital images and areas or regions ofinterest 118 from digital images with x, y, and z coordinates. Thedisplacement is then calculated for each area in the digital image, withvariable resolution, and distortion is removed or measured or otherwisequantified. Multiple digital images can be analyzed, corrected, andmosaicked all at once by a bundle adjustment, in which interrelated setsof equations are used to find a globally optimal set of correctionsacross all of a number of digital images. Spectral conditions arehandled in an analogous way by correlating light intensities fordifferent color bands and then compensating for 3D influences of thedigital model.

Diagnostic Protocol

One objective of providing normalized digital images is to enableclinicians to be able to make diagnostic decisions. Using the ABCD rulesas would be considered normal in dermatology, the tissue imaging anddigital image processing system 200 can provide data that would enableclinicians to interpret the data in a manner that provides a rapid andconsistent method of diagnostic reference. Image morphology data wouldbe available to automatically update the ABCD protocol with data aboutasymmetry, irregularity diameter and supplemental data regarding theevolving nature and form of and skin lesion.

In implementations, a vasodilator may be applied to the tissue 102(e.g., skin) or taken orally to enable the measurement of changes inblood flow in the tissue, and to assist in contrast enhancement of thearea or region of interest 118.

High Specificity Analysis

Optical analysis can be used to access the molecular components oftissue and can be used to characterize the physical changes in tissues.The tissue imaging and digital image processing system 200 may employ anoptical tissue imaging system 100 (FIG. 1), which may include a digitalcamera 104. Such an optical tissue imaging system 100 may employmonochromatic light control structures, for example a grating or bandpass filter, or a series of optical filters. Additionally, oralternatively, the optical tissue imaging system 100 (FIG. 1) may beoperated in a controlled environment, where additional metadata or otherinformation is stored in non-transitory storage media characterizing theambient conditions. For example, metadata may characterize theorientation of the subject tissue to the optical axes, the use orpresence of filters on both the irradiant source excitation and subjectradiant emission imaging axes. In the case where “off the shelf” digitalcameras 104 are used, the digital cameras 104 often have integralfilters positioned over the sensor(s) to normalize the response of thesensor in the visible portion of the electromagnetic spectrum. In somecases, the visible portion of the electromagnetic spectrum is notdesirable or the use of light in non-visible portions (e.g., UV or NIR)of the electromagnetic spectrum may advantageously enhance analysis. Inthis case, the filters could be removed or omitted to allow the maximummulti-spectral response of the wavelengths of interest.

In these manners, close control over the optical characteristics of thetissue imaging system 100 (FIG. 1) can be maintained and adjusted, andthe tissue image processing host computer system 202 can be used forhigher degrees of specificity than might otherwise be the case.Optimizing the variable conditions may assist in using the tissue imageprocessing host computer system 202 for situations typical to clinicalgeneral practice. This could also become typical of a dedicated tissueimaging and digital image processing system 200 as might be used forcolposcopy, dentistry, during surgery, or for other applications thatrequire a high degree of precision.

Communications

In practical applications, the tissue imaging system 100 (FIGS. 1 and 2)and the tissue image processing host computer 202 (FIG. 3) may beremotely located from one another, for example at different sites orfacilities. Communications of digital images can be made by comparing aseries of films and not digital data. This would require the handling ofthe physical media, but ideally the communication of images is sharedover a network, for example including the Internet. Digital images canbe transferred from a camera 104 of the tissue imaging system 100 to atissue image processing host computer 202 via the Internet and/or someother network(s). For example, digital images could be attached to anelectronic mail message (i.e., email), or communicatively transferred inan encrypted format, for example via a Web based server. Various methodsof data encryption or decryption may be used to ensure privacy as wouldbe expected in the handling of medical data. Other methods could includephysically delivering a disk or memory card to an image analysisservice. In some cases the tissue image processing host computer 202 andthe tissue imaging system 100 are in the same location and the tissueimage processing host computer 202 would then create a laboratoryimaging report 270.

Signal Enhancement

In a controlled environment, various techniques of signal enhancementcan be used such as pulsed excitation sources 106 (FIG. 1) and digitalfilters (not shown) that allow for frequency or amplitude modulation ortime filtering. For instance, detection of short time domainfluorescence would require sharp cut off of the excitation source 106.Correlation of triggering of optical excitation source 106 with sensorelectronics allows for the use with frequency or amplitude methods,including electronic methods such as FM and BPSK or mechanical shuttersand choppers. Detection of low light signals might be further enhancedby measuring and filtering the ambient signal noise.

Probability Index

In at least some embodiments, the tissue image processing host computersystem 202 uses a probability index, which is the combination ofdistributed properties resulting from combined probability analysis ofone or more variables including normalization, exposure correction,geometric correlation, color correction, signal to noisecharacterization and diagnostic protocols as would be used in a timeseries.

This can be applied using multivariate time series analysis techniquessuch as linear methods based on correlation functions or spectraldecompositions or nonlinear approaches such as recurrence features andinclude the determination of heterogeneous data in subsurface layers,layer matching and morphological image matching.

Physical means of image correction can also be used to acquire opticalmetadata prior to the image acquisition. Optical filters can be used,such as spectral bandpass filters, polarizer's, or the addition ofspectral indices that were created by using a spectrophotometer can beused to act as digital filters.

Operation is described below with respect to specific examples. It willbe understood that the following examples are intended to describevarious embodiments but are not intended to limit the embodiments in anyway.

EXAMPLES Example 1 Normalization of Excitation

In one example, the inclusion of fiducial marker(s) 116 (FIGS. 1 and 2)in digital images captured using a digital camera 104 (FIG. 1) allowsthe tissue image processing host computer 202 (FIG. 3) to compensate forvariations in apparent optical excitation such as variations inexcitation source to subject distance. The images may have been acquiredwith different digital cameras 104, and/or with variations in andspatial and spectral sensitivity. Other variations might includedistance to subject, focal length and optical axes. Ideally when digitalimages are captured, the excitation source 106 (e.g., flash) should beON or illuminating the subject tissue 102, the optical normal of boththe region of interest (e.g., lesion) and the fiducial marker 116 shouldbe equal and perpendicular to the optical normal of the digital camera104, however, this also must be accounted for. Variations might also bethe result of ambient conditions and poor image quality. The reflectionand the backscatter from the fiducial marker 116 enables the spectraldistribution of the entire image to be registered by the tissue imageprocessing host computer 202.

Example 2 Reflection vs. Backscatter

In one example, the tissue image processing host computer 202 (FIG. 3)characterizes the tissue response from various optical layers of tissue102 (FIGS. 1 and 2) in order to be able to differentiate tissuevariations that might not be fully visible to the naked eye. The spectrafrom two digital images are normalized using the fiducial marker 116and/or areas of tissue 102 in some or all digital images where melaninor hemoglobin spectra appear normal. The spectral distribution iscompared between either the whole digital image or a localized area ofthe digital image. With the major spectral components removed thatcontribute to reflection from the Epidermal Layer I₁L_(E), the changesin other optical layers such as Melanin Layer I₁L_(M), Hemoglobin LayerI₁L_(H), Collagen Layer I₁L_(C) or I₁L_(CF) can be more easily compared.

Example 3 Spatial Distribution

In one example, low cost digital cameras 104 (FIG. 1) with minimalcapacity for measuring spectral changes but adequate for accessing theshape and color or lesion borders are employed to capture digital imagesof the tissue 102. Correlation of digital images enables a clinician orautomated system (e.g., tissue image processing host computer 202 ofFIG. 3) to assess if there are morphological changes. Thesemorphological changes would—subsequently be noted in the ABCD guidanceof tissue evolution. The digital images timeline is corrected based onthe fiducial marker(s) 116 appearing in the source image I₁(x_(n), y)and the fiducial marker(s) 116 appearing within the time series imagesI_(1 . . . n)(x_(n), y_(n)).

Example 4 Controlled Laboratory System and Analysis

In one example, a patient specific model generated by a tissue imageprocessing host computer 202 (FIG. 3) allows cross reference betweendigital images in a timeline or sequence for the same patient. A usefulanalysis would determine if there is a relationship between spectralchanges whether or not there has been any noticeable change inmorphology. Once correction has been made for the optical tissue imagingsystem 100 (FIG. 1), and an image registration has been made, a timeseries of digital images can be compared. In an optical tissue imagingsystem 100 capable of being used with cut off filters, and with ambientconditions remaining similar, then regions of interest 118 (e.g., skinlesions) can be measured and a digital record of the digital imageand/or measured information stored for later reference. In a time seriesof digital images, increases in NADH fluorescence, decreases of collagenfluorescence, physical scatter of light from tissue at various physicallayers due to tissue density, spectral distribution due to size of cellnuclei, and changes in hemoglobin absorption due to increased blood flowor oxygenation may be combined in a diagnostic image layer. In somecases such digital images and associated information may be used totrack tumor formation or be used as a screening tool.

Example 5 Single Image Compared to Phantom Image

In one example, rather than correcting a single digital image formorphologic changes, is the digital image is corrected for the digitalimage's spectral distribution compared to a tissue phantom. A tissueimage processing host computer 202 (FIG. 3) breaks the digital imagedown into spectral layers. The resulting data is used as an input to theABCD rules, where as part of the color analysis, the spectraldistribution is compared to a standard and is reported as variations inthe visible, Ultra Violet, Near Infra Red, with particular notes oremphasis on: any brown or black streaks; textures variations measured byreflection; and pink or red areas. The phantom image could be a baselinestandard, could be an image taken on the subject in an area where thereis no concern, or could be based on a projection of optical features ofa fiducial marker 116 (FIGS. 1 and 2).

Example 6 Contrast Agent

In one example, a first digital image is captured before and a seconddigital image is captured after the use of a contrast agent such as adye that combines with protein or bacteria, and/or after administrationof a vasodilator, and/or a biomarker with a fluorescent marker, and/oracetic acid or water. The first and second digital images are combinedand the difference is used to screen for the tissue or region ofinterest.

CONCLUSION

The above description of illustrated embodiments, including what isdescribed in the Abstract, is not intended to be exhaustive or to limitthe embodiments to the precise forms disclosed. Although specificembodiments of and examples are described herein for illustrativepurposes, various equivalent modifications can be made without departingfrom the spirit and scope of the disclosure, as will be recognized bythose skilled in the relevant art. The teachings provided herein of thevarious embodiments can be applied to other systems, not necessarily theexemplary tissue imaging and digital image processing system generallydescribed above.

For instance, the systems, devices, and methods described herein mayalso be applied to other testing and/or analysis, including testing oranalyzing the effect of various cosmetics or therapeutics on tissue,such as skin, to discern whether such cosmetics have a beneficialeffect. For example, digital images may be captured of a region ofinterest (e.g., proximate the eyes, chin or mouth) both beforeapplication of a cosmetic or therapeutic material or treatment, andfollowing such application or treatment. The tissue image processinghost computer systems may analyze the digital images to assess theeffect of the cosmetic or therapeutic material or treatment on theregion of interest. For example, the tissue image processing hostcomputer systems 202 may determine whether a level of hydration has beenincreased, whether a level of blood flow or oxygenation has increased,whether a total number of wrinkles have decreased, or size of wrinklesdecreased, or whether blemishes have been reduced.

The system may be used to measure skin health or beauty, includingsurface and subsurface layers. For example, the system may be used tomeasure or otherwise assess a degree of hydration or a level or rate ofblood flow, which could be further compared to collagen. Alternatively,or additionally, the system may measure or otherwise assess a totalnumber of wrinkles in a given area and/or size of such wrinkles, and/ora total number of blemishes in a given area and/or size of suchblemishes.

In particular, the wavelength ratios of the optical layers could be usedto characterize a skin hydration assessment between two images atdifferent times. This could be used with a fixed optical set up thatincludes optical excitation sources located at different fixed angles tobetter access skin reflection. Such can be measured with either aphysical or projected fiducial marker scatter layer employed. Forinstance a projected marker, if it were linear and monochromatic, wouldhave variable scatter along its axes. The excitation wavelength of thisfiducial marker could be varied to include reference to other opticallayers such as the water absorption layer, the hemoglobin absorptionlayer, and the collagen absorption layer. If such a linear andmonochromatic fiducial marker were compared over a series or sequence ofdigital images, the ability to characterize the skin surface, such aswrinkles, and the skin health such as hydration, blood flow andcollagen, could be visually presented, for example as indices, graphs oras comparative images.

Such a system could be used as a standardized approach to determine thehealth and beauty impacts of various cosmetics, moisturizers,therapeutic materials, other skin creams, and/or therapeutic treatments.Such may be advantageously employed in clinical trials or for use inpoint of sale retail facilities where an individual's skin could beassessed and the indices of skin health or beauty could be used forproduct selection. Results of such an assessment may be presented in avisual form, for example a display or printout of indices, graphs orcomparative images.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, schematics,and examples. Insofar as such block diagrams, schematics, and examplescontain one or more functions and/or operations, it will be understoodby those skilled in the art that each function and/or operation withinsuch block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. The present subjectmatter may be implemented via Application Specific Integrated Circuits(ASICs). However, those skilled in the art will recognize that theembodiments disclosed herein, in whole or in part, can be equivalentlyimplemented in standard integrated circuits, as one or more computerprograms running on one or more computers (e.g., as one or more programsrunning on one or more computer systems), as one or more programsrunning on one or more controllers (e.g., microcontrollers) as one ormore programs running on one or more processors (e.g., microprocessors),as firmware, or as virtually any combination thereof, and that designingthe circuitry and/or writing the code for the software and or firmwarewould be well within the skill of one of ordinary skill in the art inlight of this disclosure.

In addition, those skilled in the art will appreciate that themechanisms taught herein are capable of being distributed as a programproduct in a variety of forms, and that an illustrative embodimentapplies equally regardless of the particular type of nontransitorysignal bearing storage media used to actually carry out thedistribution. Examples of nontransitory signal bearing storage mediainclude, but are not limited to, the following: recordable type mediasuch as floppy disks, hard disk drives, CD ROMs, digital tape, andcomputer memory; and other non-transitory computer-readable storagemedia.

The various embodiments described above can be combined to providefurther embodiments. All of the U.S. patents, U.S. patent applicationpublications, U.S. patent applications, foreign patents, foreign patentapplications and non-patent publications referred to in thisspecification and/or listed in the Application Data Sheet, includingU.S. provisional patent application Ser. No. 61/311,750, filed Mar. 8,2010, are incorporated herein by reference, in their entirety. Aspectsof the embodiments can be modified, if necessary to employ concepts ofthe various patents, applications and publications to provide yetfurther embodiments.

These and other changes can be made to the embodiments in light of theabove-detailed description. In general, in the following claims, theterms used should not be construed to limit the claims to the specificembodiments disclosed in the specification and the claims, but should beconstrued to include all possible embodiments along with the full scopeof equivalents to which such claims are entitled. Accordingly, theclaims are not limited by the disclosure.

I claim:
 1. A system for use in tissue analysis, the system comprising:at least one processor; and at least one nontransitory storage mediumthat stores processor executable instructions which when executed causethe at least one processor to: compare an appearance of at least oneshape of a virtual fiducial marker in a first digital image of a portionof a tissue to at least one defined actual shape of the virtual fiducialmarker; compare an appearance of each of a plurality of sections of aphysical fiducial marker in the first digital image to respective onesof defined sections of the physical fiducial marker, wherein the definedsections include a number of tissue phantoms each having a respectivespectral characteristic that matches a respective spectralcharacteristic of tissue of a type represented in the first digitalimage; and at least one of correlate, normalize, or correct at least thefirst digital image, based at least in part on the comparisons; thesystem being characterized in that the physical fiducial marker includesa scatter layer that overlies at least some but not all of the tissuephantoms and which simulates an optical character of the type of tissuerepresented in the first digital image, and the instructions cause theat least one processor to compare the appearance of the sections whichinclude the tissue phantoms which are overlaid by the scatter layer witha number of defined sections which include tissue phantoms that are notoverlaid by the scatter layer.
 2. The system of claim 1 wherein a numberof sections of the physical fiducial marker include a respective colorincluding at least one of black, white, a plurality of different shadesof grey, and a plurality of additional colors that are not black, whiteor grey, and the instructions cause the at least one processor tocompare the appearance of the sections which include the respectivecolors with respective ones of a defined set of respective colors. 3.The system of claim 1 wherein the instructions further cause the atleast one processor to store the digital image as a multi-layer imagefile, including a first digital image layer that stores digital imagesand at least a second digital image layer that stores image metadata. 4.The system of claim 3 wherein the instructions further cause the atleast one processor to store to a diagnostic layer of the digital imageinformation indicative of at least one of an NADH fluorescence, acollagen fluorescence, a physical scattering of light from the tissue ata number of physical layers of the tissue due to tissue density, aspectral distribution due to a size of a cell nuclei, and a hemoglobinabsorption due to increased blood flow or oxygenation; whereinnormalizing a surface reflection in each digital image in a multipleimage series, is achieved by normalizing color markers to allow accuratecomparison of a Epidermal Layer to the Melanin Layer in the in proximityto lesion and where subsurface scatter is normalized to those ofhemoglobin and collagen at proximity to lesion site border, and wherespectral distribution of borders of the lesion has been established, thespectral distribution of the borders is compared to the surroundingtissue and a lesion map generated.
 5. The system of claim 3 wherein theinstructions further cause the at least one processor to register anumber of subsequent digital images in spatial and optical relationshipand to compare the first and the subsequent digital images on a layer bylayer basis.
 6. The system of claim 1 wherein the instructions furthercause the at least one processor to reference at least one of spectralmeasurements or optical density at specific coordinates in the firstdigital image to allow later comparison to measurements in a number ofsubsequent digital images of a region of interest.
 7. The system ofclaim 1 wherein the instructions further cause the at least oneprocessor to compare a number of ratios of respective radiant spectralintensity of a number of wavelengths or wavebands in the first digitalimage.
 8. The system of claim 7 wherein the instructions further causethe at least one processor to compare the number of ratios of respectiveradiant spectral intensity of the number of wavelengths or wavebands inthe first digital image to a number of ratios of a respective radiantspectral intensity of a number of wavelengths or wavebands in at leastone subsequent digital image.
 9. The system of claim 1 wherein theinstructions further cause the at least one processor to normalize aplurality of digital images including the first digital image bymeasuring a difference of a spectral distribution between an opticalcharacter of the tissue in comparison to the physical fiducial marker,where a linear relationship between a number of defined spectralrelationships is within or exceeds a limit of a normal spectraldistribution.
 10. The system of claim 1 wherein the instructions furthercause the at least one processor to establish a subject specificbaseline which is specific to an individual, and normalize based atleast in part on the subject specific baseline the first digital imageand a plurality of sequential digital images, the sequential digitalimages sequentially captured at various times following a capture of thefirst digital image; wherein normalizing a surface reflection in eachdigital image in a multiple image series, is achieved by normalizingcolor markers to allow accurate comparison of a Epidermal Layer to theMelanin Layer in the in proximity to lesion and where subsurface scatteris normalized to those of hemoglobin and collagen at proximity to lesionsite border, and where spectral distribution of borders of the lesionhas been established, the spectral distribution of the borders iscompared to the surrounding tissue and a lesion map generated.
 11. Thesystem of claim 10 wherein the instructions further cause the at leastone processor to determine differences in a region of interest as theregion of interest appears between the normalized digital imagesincluding the first digital image and the plurality of sequentialdigital images as part of an analysis.
 12. The system of claim 11wherein the instructions further cause the at least one processor todetermine morphological changes of the region of interest as the regionof interest appears between the digital images as part of thedetermination of the differences in the region of interest as the regionof interest appears between the normalized digital images including thefirst digital image and the plurality of sequential digital images. 13.The system of claim 11 wherein the instructions cause the at least oneprocessor to determine the differences by assessing any change in atleast one of a level of skin hydration, a total number of wrinkles or asize of at least one wrinkle, or a total number of blemishes or a sizeof at least one blemish.
 14. The system of claim 11 wherein theinstructions cause the at least one processor to determine thedifferences by assessing at least one of a level of hydration or a levelof blood flow between the first digital image and at least onesubsequent digital image, where the first digital image represents theregion of interest prior to a first application of a cosmetic, amoisturizer, a therapeutic or a therapeutic treatment and the at leastone subsequent digital image represents the region of interest after thefirst application of the cosmetic, the moisturizer, the therapeutic orthe therapeutic treatment.
 15. The system of claim 1 wherein theinstructions further cause the at least one processor to normalize atleast the first digital images based at least in part on a spectralmarker of hemoglobin and a spectral marker of collagen.
 16. The systemof claim 1 wherein the instructions further cause the at least oneprocessor to generate a probability index based on a combination ofdistributed properties of a number of variables including anormalization, an exposure correction, a geometric correlation, anoptical spectroscopic correction, a signal to noise characterization, ora defined diagnostic protocol.
 17. The system of claim 1 wherein theinstructions further cause the at least one processor to generate adigital model that enables enhancement of a region of interest for athree dimensional model based on spatial and spectral data from thedigital images.
 18. The system of claim 17 wherein the instructionsfurther cause the at least one processor to associate at least one ofmultispectral data or image timeline data to the digital model thatgeometrically represents the region of interest in three dimensions. 19.The system of claim 17 wherein the instructions further cause the atleast one processor to rectify the tissue with a three dimensional mapof at least a portion of a body which combines a set of threedimensional model probabilities with a correlation of a set ofcoordinate locations, a set of spectral effects and a set of complexinteractions.
 20. The system of claim 1 wherein the instructions furthercause the at least one processor to correct at least the first digitalimage based at least in part on color correction information.
 21. Thesystem of claim 20 wherein the instructions further cause the at leastone processor to generate a digital multidimensional lesion map, wherebythe digital lesion map is generated by tracking a set of pixelcharacteristics in at least the first digital image including at leastone of a surface, a sub-surface, other layers or a depth characteristicof the tissue as determined from a spectral analysis of the tissue asrepresented in at least the first digital image.
 22. The system of claim20 wherein the instructions further cause the at least one processor tocorrect for spectral effects in the tissue represented in at least thefirst digital image which spectral effects are due to interactions oflight absorption, reflectance and fluorescence, and to cross referenceand compare a number of spatial and a number of spectral componentsspecified by at least one of a digital model of tissue image data oranother digital image to generate a digital three dimensional model of aregion of interest.
 23. The system of claim 20 wherein the instructionsfurther cause the at least one processor to correct for differences inspatial orientation of at least one of an excitation axis or an imagingaxis of a tissue imaging system in Cartesian space, and wherein theprocessor models tissue image data in order to cross reference andcompare data from various spectral components and/or from differentdigital images.
 24. The system of claim 1 wherein the instructionsfurther cause the at least one processor to perform a registration oneach of a plurality of digital images of the tissue, including the firstdigital image, based at least in part on a variation between image layercoordinates in a temporal sequence of a plurality of digital images ofthe tissue.
 25. The system of claim 1 wherein the instructions furthercause the at least one processor to generate an analysis comparison oflayers in at least the first digital image as a histogram.
 26. Thesystem of claim 1 wherein the instructions further cause the at leastone processor to generate a probability distribution of a tissue beingabnormal.
 27. The system of claim 22 wherein the instructions furthercause the at least one processor to generate a probability distributionof the tissue being abnormal based at least in part on a comparison ofan optical density to a percentage of optical density that isattributable to collagen.
 28. The system of claim 1 wherein theinstructions further cause the at least one processor to generate anabnormal relationship between each of a plurality of digital images,wherein the images are viewed within a probability index that weights atleast some digital images according to at least one of a diagnosticvalue or a comparative amount of change between spectra.
 29. A system asin claim 1, wherein the executable instructions when executed furthercause the at least one processor to: assess a change in at least one ofa level of hydration, a level of blood flow, a total number of wrinkles,a size of at least one wrinkle, a total number of blemishes, or a sizeof at least one blemish between the first digital image of a region ofinterest of the tissue and at least one subsequent digital image of theregion of interest of the bodily tissue, where the first digital imagerepresents the region of interest prior to a first application of acosmetic, a moisturizer, a therapeutic or a therapeutic treatment andthe at least one subsequent digital mage represents the region ofinterest after the first application of the cosmetic, the moisturizer,the therapeutic or the therapeutic treatment; and report the assesseddifference in a visual form.
 30. The system of claim 29 wherein thechange is assessed by assessing a number of spectral characteristics ofthe region of interest in the first and the at least one subsequentdigital image.
 31. The system of claim 30 wherein the instructions causethe at least one processor to assess a number of spectralcharacteristics of the region of interest in the first and the at leastone subsequent digital image by determining a spectral absorption,reflectance or fluorescence response of a number of layers of skincharacteristic of water, hemoglobin, and collagen.
 32. The system ofclaim 30 wherein the at least one processor performs the assessing afterperforming the at least one of correlation, normalization, or correctionof at least the first digital image based at least in part on thecomparisons.
 33. The system of claim 30 wherein the at least oneprocessor includes a therapy recommendation in the report based on theassessment.